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Abstract 

Experimental data have revealed that neuronal connection efficacy exhibits two forms 
of short-term plasticity, namely, short-term depression (STD) and short-term facilitation 
(STF). They have time constants residing between fast neural signaling and rapid learn- 
ing, and may serve as substrates for neural systems manipulating temporal information on 
relevant time scales. The present study investigates the impact of STD and STF on the dy- 
namics of continuous attractor neural networks (CANNs) and their potential roles in neural 
information processing. We find that STD endows the network with slow-decaying plateau 
behaviors — the network that is initially being stimulated to an active state decays to a silent 
state very slowly on the time scale of STD rather than on the time scale of neural signaling. 
This provides a mechanism for neural systems to hold sensory memory easily and shut off 
persistent activities gracefully. With STF, we find that the network can hold a memory trace 
of external inputs in the facilitated neuronal interactions, which provides a way to stabilize 
the network response to noisy inputs, leading to improved accuracy in population decoding. 
Furthermore, we find that STD increases the mobility of the network states. The increased 
mobility enhances the tracking performance of the network in response to time- varying stim- 
uli, leading to anticipative neural responses. In general, we find that STD and STF tend to 
have opposite effects on network dynamics and complementary computational advantages, 
suggesting that the brain may employ a strategy of weighting them differentially depending 
on the computational purpose. 
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1 Introduction 



Experimental data have consistently revealed that the neuronal connection weight, which mod- 
els the efficacy of the firing of a pre-synaptic neuron in modulating the state of a post-synaptic 
one, varies on short time scales, ranging from hundreds to thousands of milliseconds (see, e.g.. 



(Stevens and Wang 1995; Markram and Tsodyks 1996 iDobrunz and Stevens 1997} Markram, Wang, and Tsodyk; 
This is called short-term plasticity (STP). Two types of STP, with opposite effects on the con- 
nection efficacy, have been observed. They are short-term depression (STD) and short-term 
facilitation (STF). STD is caused by the depletion of available resources when neurotransmit- 
ters are released from the axon terminal of the pre-synaptic neuron during signal transmission 
(Stevens and Wang 1995; Markram and Tsodyks 1996; Dayan and Abbott 2001). On the other 
hand, STF is caused by the influx of calcium into the presynaptic terminal after spike generation, 
which increases the probability of releasing neurotransmitters. 

Computational studies on the impact of STP on network dynamics have strongly suggested 
that STP can play many important roles in neural computation. For instance, cortical neurons 
receive pre-synaptic signals with firing rates ranging from less than 1 to more than 200 Hertz. It 
was suggested that STD provides a dynamic gain control mechanism that allows equal fractional 
changes on rapidly and slowly firing afferents to produce post-synaptic responses, realizing We- 



ber's law ([Tsodyks and Markram 1997} [Abbott, Varela, Sen, and Nelson 1997[>. Besides, com 



putations can be performed in recurrent networks by population spikes in response to external in- 



puts, which are enabled through STD by recurrent connections (Tsodyks, Uziel, and Markram 2000 



Loebel and Tsodyks 2002). 



Another role played by synaptic depression was proposed by Levina, Herrmann, and Giesel 
(2007). In neuronal systems, critical avalanches are believed to bring about optimal compu- 
tational capabilities, and are observed experimentally. Synaptic depression enables a feedback 
mechanism so that the system can be maintained at a critical state, making the self-organized 



critical behavior robust (Levina, Herrmann, and Giesel 2007). Herding, a computational algo- 



rithm reminiscent of the neuronal dynamics with synaptic depression, was recently found to have 



a similar effect on the complexity of information processing (Welling 2009). STP was also re- 
cently thought to play a role in the way a neuron estimates the membrane potential information 



of the pre-synaptic neuron based on the spikes it receives (Pfister, Dayan, and Lengyel 2010). 
Concerning the computational significance of STF, a recent work proposed an interesting idea 



for achieving working memory in the prefrontal cortex (Mongillo, Barak, and Tsodyks 2008). 
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The residual calcium of STF is used as a buffer to facilitate synaptic connections, so that inputs 
in a subsequent delay period can be used to retrieve the information encoded by the facilitated 
synaptic connections. The STF-based memory mechanism has the advantage of not having to 
rely on persistent neural firing during the time the working memory is functioning, and hence 
is energetically more efficient. 

Prom the computational point of view, the time scale of STP resides between fast neural 
signaling (in the order of milliseconds) and rapid learning (in the order of minutes or above), 
which is the time scale of many important temporal processes occurring in our daily lives, such 
as the passive holding of a temporal memory of objects coming into our visual field (the so-called 
iconic sensory memory) , or the active use of the memory trace of recent events for motion control. 
Thus, STP may serve as a substrate for neural systems manipulating temporal information on 
the relevant time scales. STP has been observed in many parts of the cortex, and also exhibits 
large diversity in different cortical areas, suggesting that the brain may employ a strategy of 
weighting STD and STF differently depending on the computational purpose. 

In the present study, we explore the potential roles of STP in processing information de- 
rived from external stimuli, an issue of fundamental importance yet inadequately investigated 
so far. For ease of exposition, we use continuous attractor neural networks (CANNs) as our 
working model, but our main results are qualitatively applicable to general cases. CANNs are 
recurrent networks that can hold a continuous family of localized active states (jAmari 1977p . 
Neutral stability is a key property of CANNs, which enables neural systems to update mem- 
ory states easily and to track time-varying stimuli smoothly. CANNs have been successfully 



applied to describe the generation of persistent neural activities (Wang 2001), the encoding of 



continuous stimuli such as the orientation, the head direction and the spatial location of objects 



(Ben-Yishai, Lev Bar-Or, and Sompolinsky 1995: Zhang 1996; Samsonovich and McNaughton 1997), 



and a framework for implementing efficient population decoding (Deneve, Latham, and Pouget 1999). 

When STP is included in a CANN, the dynamics of the network is governed by two time 
scales. The time constant of STP is much slower than that of neural signaling (100-1000 ms vs. 
1-10 ms). The interplay between the fast and the slow dynamics causes the network to exhibit 
rich dynamical behaviors, laying the foundation for the neural system to implement complicated 
functions. 

In CANNs with STD, various intrinsic behaviors have been reported, including damped oscil- 



lations (Tsodyks, Pawelzik, and Markram 1998), periodic and aperiodic dynamics (Tsodyks, Pawelzik, and Markre 
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state hopping with transient population spikes (Holcman and Tsodyks 2006), travehng fronts 



and pulses (IPinto and Ermentrout 2001t BresslofF, Folias, Prat, and Li 2003 IFolias and Bressloff 2004^ 



Kilpatrick and Bressloff 2010), breathers and pulse-emitting breathers (Bressloff, Folias, Prat, and Li 2003 



IFolias and Bressloff 2004|) . spiral waves (Kilpatrick and Bressloff 2009), rotating bump states 



([York and van Rossum 20"09t Igarashi, Qizumi, Otsubo, Nagata, and Okada 2009 ) , and self-sustained 



non-periodic activities (jStratton and Wiles 2010]) . Here, we focus on those network states rel- 
evant to the processing of stimuli in CANNs, including static, moving and metastatic bumps 
(|Wu and Amari 2005 [ |Fung, Wong, and Wu 2010"| ). More significantly, we find that with STD, 



the network state can display slow-decaying plateau behaviors, i.e., the network that is initially 
being stimulated to an active state by a transient input decays to the silent state very slowly on 
the time scale of STD relaxation, rather than on the time scale of neural signaling. This is a very 
interesting property. It implies that STD can provide a way for the neural system to maintain 
sensory memory for a duration unachievable by the signaling of single neurons, and shut off the 
network activity of sensory memory naturally. The latter has once been a challenging technical is- 



sue in the study of theoretical neuroscience (Gutkin, Laing, Colby, Chow, and Ermentrout 2001). 



With STF, neuronal connections become strengthened during the presence of an external 
stimulus. This stimulus-specific facilitation lasts for a period on the time scale of STF, and pro- 



vides a way for the neural system to hold a memory trace of external inputs ( Mongillo, Barak, and Tsodyks 2008 ) . 
This information can be used by the neural system for various computational tasks. To demon- 
strate this idea, we consider CANNs as a framework for implementing population decoding 



(Deneve, Latham, and Pouget 1999; Wu, Amari, and Nakahara 2002). In the presence of STF 



the network response is determined not only by the instant input value but also by the history 
of external inputs (the latter being mediated by the facilitated neuronal interactions). There- 
fore, temporal fluctuations in external inputs can be largely averaged out, leading to improved 
decoding results. 



In general, STD and STF tend to have opposite effects on network dynamics (Torres, Cortes, Marro, and Kappe 
The former increases the mobility of network states, whereas the latter increases their stabil- 
ity. Enhanced mobility and stability can contribute positively to different computational tasks. 
Enhanced stability mediated by STF can improve the computational and behavioral stabil- 
ity of CANNs. To demonstrate that enhanced mobility does have a positive role in informa- 
tion processing, we investigate a computational task in which the network tracks time-varying 
stimuli. We find that STD increases the tracking speed of a CANN. Interestingly, for strong 
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STD, the network state can even overtake the moving stimulus, reminiscent of the anticipa- 
tive responses of head-direction and place cells (Blair and Sharp 1995 : lO^'Keefe and Recce 1993t 



Romani and Tsodyks 20111 ) . 

The rest of the paper is organized as follows. After introducing the models and methods in 
Section 2, we discuss the intrinsic properties of CANNs in the absence of external stimuli by 
studying their phase diagram in Section 3. In Sections 4 to 6, we study the network behavior in 
the presence of various stimuli. In Section 4, we consider the after-effects of a transient stimulus, 
and find that sensory memories can persist for a desirable duration and then decay gracefully. 
In Section 5, we consider the response of the network to a noisy stimulus, and find that the 
accuracy in population decoding can be enhanced. In Section 6, we consider the response of 
the network to a moving stimulus, and find that the tracking performance is improved by the 
enhanced mobility of the network states. The paper ends with conclusions and discussions in 
Section 7. Our preliminary results on the effects of STD have been reported in Fung, Wong, 
Wang, and Wu (2010). 

2 Models and Methods 

We consider a one-dimensional continuous stimulus x encoded by an ensemble of neurons. For 
example, the stimulus may represent a moving direction, an orientation or a general continuous 
feature of objects extracted by the neural system. We consider the case where the range of 
possible values of the stimulus is much larger than the range of neuronal interactions. We can 
thus effectively take x G (— c«,oo) in our analysis. In simulations, however, we will set the 
stimulus range to be —L/2 < x < L/2, and have N neurons uniformly distributed in the range 
obeying a periodic boundary condition. 

Let u{x, t) be the current at time t in the neurons whose preferred stimulus is x. The 
dynamics of u{x,t) is determined by the external input the network input from other 

neurons, and its own relaxation. It is given by 
du{x, t) 



Is 



dt 



{x, t) + P^\x, t)+p dx'J{x, x')p{x', t) [I + fix', t)] r{x', t), (1) 



where Tg is the synaptic time constant, which is typically of the order of 2 to 5 ms, and p the 
neural density. r{x, t) is the firing rate of neurons, which increases with the synaptic input, but 
saturates in the presence of global activity-dependent inhibition. A solvable model that captures 
these features is given by Deneve, Latham, and Pouget (1999) and Wu, Amari, and Nakahara 



(2002) 

( t) Cr^('^; 0^ ^2^^ 

' 1 + kp J_^dx'u{x' ,ty^ 
where k is a positive constant controlling the strength of global inhibition and Cr is a constant 
whose dimension is r (x, t) u (x, This type of global inhibition can be achieved by shunting 
inhibition ( [Heeger 1992t |Hao, Wang, Dan, Poo, and Zhang 2009[ ). 



J(x, x') is the baseline neural interaction from x' to x when no STP exists. In our solvable 
model, we choose J{x,x') to be of the Gaussian form with an interaction range a, i.e., 



J{x,x ) = — j= exp 
av27r 



where Jo is a constant. J{x, x') is translationally invariant, in the sense that it is a function of 
X — x' rather than x or x' . This is the key to generating the neutral stability of CANNs. 

The variable p{x, t) represents the pre-synaptic STD effect, which has the maximum value 



of 1 and decreases with the firing rate of the neurons (Tsodyks, Pawelzik, and Markram 1998 
Zucker and Regehr 2002| . Its dynamics is given by 



Td^^^ = 1 - p{x, t) - TdPp{x, t)[l + f{x, t)]r{x, t), (4) 

where Td is the time constant for synaptic depression, and the parameter j3 controls the depression 
effect due to neuronal firing. 

The variable f{x,t) represents the pre-synaptic STF effect, which increases with the firing 
rate of the neurons but saturates at a maximum value /max- Its dynamics is given by 

^ df{x^t) ^ ^) _^ ^^Q, [j^^^ _ ^(3,^ (5) 

where Tf is the time constant for synaptic facilitation, and the parameter a controls the facili- 
tation effect due to neuronal firing. 

The dynamical equations (jH and ([5]) are consistent with the phenomenological models of STD 



and STF fitted to experimental data (Tsodyks, Pawelzik, and Markram 1998 ) (see Appendix A) 



From Eqs.([l]) and ([5]), we can calculate the steady state values of p and /, which are 

1 + Tfar 

Hence, in the high frequency limit, Tjar ^> 1, we have / /max, which can be regarded as a 
constant, and p = 1/ [1 -|- Td/3{1 + /max)?']- In this case, we only have to consider the effect of 



STD. In the low frequency limit, Trf/3(1 + f)r <^ 1, and so p ^ 1 and we only need to consider the 
effect of STF. Note, however, that the terms "high frequency limit" and "low frequency limit" 
are used figuratively. The actual limits should depend on the other parameters mentioned above. 

Our theoretical analysis of the network dynamics is based on the observations that: 1) the 
stationary states of the network, as well as the profile of STP across all neurons, can be well 
approximated as Gaussian-shaped bumps; and 2) the state change of the network, and hence the 
profile of STP, can be well described by distortions of the Gaussian bump in various forms. We 
can therefore use a perturbation approach developed by Fung, Wong, and Wu (2010) to solve 
the network dynamics analytically. 

It is instructive for us to first review the network dynamics when no STP is included. This 
is done by setting /3 = in Eq. ^ and a = in Eq. ([5]), so that p(x, t) = 1 and f{x, t) = for 
all t. In this case, the network can support a continuous family of stationary states when the 
global inhibition is not too strong. These steady states are 



U(X z) 



rixizj 



uq exp 



ro exp 



4a2 
(x - 
2a2 



(8) 
(9) 



where uq = [1 + (1 - k/kc)^/"^] (CrJo) /(iak^), ro = [1 + (1 - k/kcy/'^]/{2akpV2^) and kc = 
p (Cr Jo)^ /(8a\/27r). These stationary states are translationally invariant among themselves and 
have the Gaussian shape with a free parameter z representing the peak position of the Gaussian 
bumps. They exist for < k < kc, and kc is the critical inhibition strength above which only 
silent states with uq = exist. 

Because of the translational invariance of the neuronal interactions, the dynamics of CANNs 
exhibits unique features. Fung, Wong, and Wu (2010) have shown that the wave functions of the 
quantum harmonic oscillators can well describe the different distortion modes of a bump state. 
For instance, during the process of tracking an external stimulus, the synaptic input u{x, t) can 
be written as 

oo 

u{x, t) =^ anit)Vn{x\z{t)), (10) 
n=0 

where Vn{x\z{t)) are the wavefunctions of the quantum harmonic oscillator (Fig. [1]), 



, , , (-l)«(V2a)"-i/2 
Vn[x z) = exp 

V7HV2ri!2" 



{x — zY 
4a2 



d 

Tx^ 



(x — z) 
2^ 



2n 



n 



0,1,.... (11) 



These functions have clear physical meanings, corresponding to distortions in the height, posi- 
tion, width, skewness and other higher order features of the Gaussian bump (see Fig. [1]). We 



Height Position Width Skewness 




XXX 



Figure 1: (a) - (d) The first distortion modes of tlie bump state, (e) - (li) Their effects of 
producing distortions, respectively, in the height, position, width and skewness of the Gaussian 
bump. Solid and dashed lines represent distorted and undistorted bumps respectively. 



can use a perturbation approach to solve the network dynamics effectively, with each distortion 
mode characterized by an eigenvalue determining its rate of evolution in time. A key property of 
CANNs is that the translational mode has a zero eigenvalue, and all other distortion modes have 
negative eigenvalues for k < kc- This implies that the Gaussian bumps are able to track changes 
in the position of the external stimuli by continuously shifting the position of the bumps, with 
other distortion modes affecting the tracking process only in the transients. An example of the 
tracking process is shown in Fig. [2l where we consider an external stimulus with a Gaussian 
profile given by 



r''\x,t) = Aexp 



(x-zit)) 



21 



(12) 



4a2 

The stimulus is initially centered at z = 0, pinning the center of a Gaussian neuronal response 
at the same position. At time t = 0, the stimulus shifts its center from zq = to zq = 3a 
abruptly. The bump moves towards the new stimulus position, and catches up with the shift of 
the stimulus after a certain time, referred to as the reaction time. 

We can generalize the perturbation approach developed by Fung, Wong, and Wu (2010) 
to study the dynamics of CANNs with dynamical synapses. We will only present the detailed 
analysis to the case of STD. Extension to the case of STF is straightforward. 

Similar to the synaptic input u{x,t), the profile of STD can be expanded in terms of the 
distortion modes, 

oo 

p{x,t) = l-^bn{t)Wn{x\z{t)), (13) 
n=0 
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(a) 




Figure 2: (a) The neural response profile tracks the change in position of the external stimulus 
from zo/a = to 3 at t = 0. Parameters: a = 0.5, k = 0.95, ^ = 0.0085, pCrJoA = 4.82843. (b) 
The profile of u{x,t) at t/r = 0, 1, 2, • • • ,10 during the tracking process in (a). 



where Wnixlz) is given by 



Wnix\z) 



;-i)"a"-i/2 

\/7ri/2n!2" 



exp 



(x - z)' 
2a2 



{x - zf 



(14) 



Note that the width of Wn {x\z') is \pl times that of w„ {x\z) due to the appearance of r (x, t) oc 
u (x, t)^ in Eq. ©. 

Substituting Eqs. (jlOp and (jl3p into ([T|) and and using the orthonormality and com- 
pleteness of the distortion modes, we get the dynamical equations for the coefficients a„(t) and 
bn{t). The details are presented in Appendix B. 

The peak position z{t) of the bump is determined from the self-consistent condition, 

dxu{x, t)x 



z{t) 



f oo 

-oo ■ 

POO 



(15) 



f^^dxu{x,t) ' 

Truncating the perturbation expansion at increasingly high orders corresponds to the inclu- 
sion of increasingly complex distortions, and hence provides increasingly accurate descriptions 
of the network dynamics. As confirmed in the subsequent sections, the perturbative approach 
is in excellent agreement with simulation results. The agreement is especially remarkable when 
the STD strength is weak, and the lowest few orders are already sufficient to explain the dy- 
namical features. The agreement is less satisfactory when STD is strong, and the perturbative 
approach typically over-estimates the stability of the moving bump. This is probably due to the 
considerable distortion of the Gaussian profile of the synaptic depression when STD is strong. 



9 



3 Phase Diagrams of CANNs with STP 



We first study the impact of STP on the stationary states of CANNs when no external input is 
appHed. For the convenience of analysis, we will explore the effects of STD and STF separately. 
This corresponds to the limits of high or low neuronal firing frequencies, or the cases where only 
one type of STP dynamics is significant. 

3.1 The Phase Diagram of CANNs with STD 

We set a = in Eq. ([5]) to turn off STF. In the presence of STD, CANNs exhibit new and 
interesting dynamical behaviors. Apart from the static bump state, the network also supports 
spontaneously moving bump states. Examining the steady state solutions of Eqs. ([T]) and 
we find that uq has the same dimension as pCrJoUQ, and 1 —p{x,t) scales as t^/Suq. Hence we 
introduce the dimensionless parameters k = k/k^ and j3 = Tdl3/{p^ (CrJo)'^)- The phase diagram 
obtained by numerical solutions to the network dynamics is shown in Fig. [3l 



ICQ. 




k 



Figure 3: Phase diagram of the network states with STD. Symbols: numerical solutions. Dashed 
line: Eq. (12ip . Dotted line: Eq. (I24p . Solid line: Gaussian approximation using the 11**^ order 
perturbation of the STD coefficient. Point P: the working point for Figs. [6] and [71 Parameters: 
Td/Ts = 50, a = 0.5/6, range of the network x G [— 7r,7r). 

We first note that the synaptic depression and the global inhibition play the same role in 
reducing the amplitude of the bump states. This can be seen from the steady state solution of 
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u{x,t), 

I' , pj(x — x')u{x'Y , N 

^ ' J l + kpj dx"u{x"Y + Tdl3u{x'Y ^ ' 

The third term in the denominator of the integrand arises from STD, and plays the role of a 

local inhibition that is strongest where the neurons are most active. Hence we see that the silent 

state with u[x^ t) = is the only stable state when either A: or /3 is large. 

When STD is weak, the network behaves similarly to CANNs without STD, that is, the 

static bump state is present up to k near 1. However, when /3 increases, a state with the 

bump spontaneously moving at a constant velocity comes into existence. Such moving states 

have been predicted in CANNs (|York and van Rossum 2009 1 Kilpatrick and Bressloff 2010 ), 



and may be associated with the travelling wave behaviors widely observed in the neocor- 



tex ( Wu, Huang, and Zhang 2008 ). At an intermediate range of /3, the static and moving states 
coexist, and the final state of the network depends on the initial condition. As /3 increases 
further, static bumps disappear. In the limit of high /3, only the silent state is present. Below, 
we will use the perturbation approach to analyze the network dynamical behaviors. 

3.1.1 Zeroth Order: The Static Bump 

The zeroth order perturbation is applicable to the solution of the static bump, since the profile 
of the bump remains effectively Gaussian in the presence of synaptic depression. Hence, when 
STD is weak and for a <^ L, we propose the following Gaussian approximations. 



{x - zf 



4a2 
(x - zf^ 



2a? 



u{x,t) = uo(t)exp 

p{x,t) = l-po{t)exp 

As derived in Appendix C, the dynamical equations for uq and po are given by 
du{t) _ u{tf 



dt 



Td- 



d pojt) 
dt 



V2il + ku{ty / 8) 
1 + fcu(t)2/8 



1 



u{t) + A, 



1 



Po(t), 



(17) 
(18) 



(19) 
(20) 



where u = pCrJoUQ is the dimensionless bump height and A = pCrJoA is the dimensionless 
stimulus strength. For A = 0, the steady state solution of u and po and its stability against 
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fluctuations of u and po are described in Appendix C. We find that stable solutions exist when 



13 < P'^' 



4(1 - ^2/3^0 



1 + 



2/3po) 



(21) 



when Po is the steady state solution of Eqs. ([T9]) and (pO]) . The boundary of this region is 
shown as a dashed line in Fig. [3l Unfortunately, this line is not easily observed in numerical 
solutions since the static bump is unstable against fluctuations that are asymmetric with respect 
to its central position. Although the bump is stable against symmetric fluctuations, asymmetric 
fluctuations can displace its position and eventually convert it to a moving bump. This will be 
considered in the flrst-order perturbation in the next subsection. 



3.1.2 First Order: The Moving Bump 

When the network bump is moving, the proflle of STD is lagging behind due to its slow dynamics, 
and this induces an asymmetric distortion in the profile of STD. Fig. |1] illustrates this behavior. 
Comparing the static and moving bumps shown in Fig. and (b), one can see that the profile 
of a moving bump is characterized by the synaptic depression lagging behind the moving bump. 
This is because neurons tend to be less active in the locations of low values of p{x,t), causing 
the bump to move away from the locations of strong synaptic depression. In turn, the region of 
synaptic depression tends to follow the bump. However, if the time scale of synaptic depression 
is large, the recovery of the synaptic depressed region will be slowed down, and the region will 
be unable to catch up with the bump motion. Thus, the bump will start moving spontaneously. 
This is the same cause attributed to anticipative non-local events modeled in neural systems 
dBlair and Sharp 1995[ [O'Keefe and Recce 1993t |Romani and Tsodyks 201ip. 



Stationai'y 



Moving 





Figure 4: Neuronal input u{x, t) and the STD coefficient p{x, t) in (a) the static state at 
(k,P) = (0.9,0.005), and (b) the moving state at (k,^) = (0.5,0.015). Parameter: Td/r^ = 50. 



To incorporate this asymmetry into the network dynamics, we consider the first-order per- 
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turbation. However, to facilitate our analysis, we make a further simplification as follows: 

(x - vt)'^' 



u{x, t) 
p{x,t) 



uo{t) exp 



1 -po{t)exp 



4a2 
(x - vtf 
2^ 



+ Piit) exp 



(x — vt) 
2^ 



2n 



X — vt 



(22) 
(23) 



This means that we have restricted the bump profile to the zeroth order. Comparison with the 
full first-order perturbation shows that the discrepancy is not significant. This is because the 
synaptic interactions among the neurons effectively maintain the bump profile in a Gaussian 
shape, whereas the STD profile is much more susceptible to asymmetric perturbations. 

As described in Appendix D, we obtain four steady-state equations for u/B, po, pi and VTg/a 
in terms of the parameters l3vP /B and Ts/t^i, where B = l + is the global inhibition factor. 

It is easy to first check if the static bump obtained in Appendix D is also a valid solution by 
setting V and pi to 0. We can then study the stability of the static bump against asymmetric 
fluctuations. This is done by introducing small values of pi and vTg/a into the static bump 
solution and considering how they evolve. As shown in Appendix D the static bump becomes 
unstable when 



1 + kW 



< Q 



(24) 



where Q = 7^7/4, R = {7 /A)[{b/2)^J7 /& - 1], and S = (343/36)(l - ^677)- This means 
that in the region bounded by Eqs. (j2ip and (j24p . the static bump is unstable to asymmetric 
fluctuations. It is stable (or more precisely, metastable) when it is static, but once it is pushed 
to one side, it will continue to move along that direction. We call this behavior metastatic. As 
we shall see, this metastatic behavior is the cause of the enhanced tracking performance. 

Next, we consider solutions with non- vanishing pi and v. We find that real solutions exist 
only if condition (j24p is satisfied. This means that as soon as the static bump becomes unstable, 
the moving bump comes into existence. As shown in Fig. [Sj the boundary of this region 
effectively coincides with the numerical solution of the line separating the static and moving 
phases. In the entire region bounded by Eqs. (f2TI) and (|24]) . the moving and (meta)static bumps 
coexist. 

We also find that when t^/ts increases, the moving phase expands at the expense of the 
(pure) static phase. This is because the recovery of the synaptic depressed region becomes 
increasingly slow, making it harder for the region to catch up with the changes in the bump 
motion, hence sustaining the bump motion. 
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3.2 The Phase Diagram of CANNs with STF 

We set /3 = in Eq. (jH) to turn off STD. Compared with STD, STF has quahtatively the opposite 
effect on the network dynamics. When an external perturbation is apphed, the dynamical 
synapses will not push the neural bump away. Instead they will try to pull the bump back 
to its original position. The phase diagram in the space of k and the rescaled STF parameter 
a = Tfa/{p'^ (CrJo)'^) is shown in Fig. [5l When a increases, the range of inhibitory strength k 
allowing for a bump state is enlarged. Note that since STF tends to stabilize the bump states 
against asymmetric fluctuations, no moving bumps exist. The phase boundary of the static 
bump is well predicted by the second-order perturbation. 
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Figure 5: Phase diagram of CANNs in the presence of STF. As the synaptic facilitation is present, 
the range allowed to support a stationary bump is broadened. Dashed line: Prediction by 0**^ 
order approximation. Solid line: Prediction by 2"'^ order approximation. Symbols: Simulations. 
Parameters: N/L = 80/(27r), a/L = 0.5/(27r), tj/ts = 50 and /max = 1- 

Concerning the time scale of neural information processing, it should be noted that it takes 
time of the order of tj for neuronal interactions to be fully facilitated. In the parameter range 
of > 1 where the facilitated neuronal interaction is necessary for holding a bump state, we 
need to present an external input for a time up to the order of r/ before a bump state can be 
sustained. 
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4 Memories with Graceful Degradation in CANNs with STD 



We consider the response of the network to a transient stimulus given by 



Text 



{x, t) = A (t) exp 



4a2 



(25) 



Here A {t) is non-zero for some duration before t = 0, so that a bump is rapidly formed, but 
A{t) vanishes after t = 0. 

The network dynamics displays a very interesting behavior in the marginally unstable region 
of the static bump. In this regime, the static bump solution barely loses its stability. The 
bump is stable if the level of synaptic depression is low, but unstable at high levels. Since 
the STD time scale is much longer than the synaptic time scale, a bump can exist before the 
synaptic depression becomes effective. This maintains the bump in the plateau state with a 
slowly decaying amplitude, as shown in Fig. ^a). After a time duration of the order of r^, 
the STD strength becomes sufficiently significant, as shown in Fig. [6l^b), and the bump state 
eventually decays to the silent state. 

0.05| 
^ 0.04- 




Figure 6: The height of the bump decays over time for two initial conditions of types A and B 
in Fig. [7] at {k,f3) = (0.95,0.0085) (point P in Fig. [3]). Symbols: numerical solutions. Dashed 
Lines: 1*^* order perturbation using Eqs. ()19l) and (I20p . Solid lines: 2"^^^ order perturbation. 
Other parameters: t^/ts = 50, a = 0.5 and x G [— 7r,7r). 



4.1 First Order: Trajectory Analysis 

It is instructive to analyze the plateau behavior first by using the first-order perturbation. We 
select a point in the marginally unstable regime of the silent phase, that is, in the vicinity of 
the static phase. As shown in Fig. [TJ the nullclines of u and po (du/dt = and dpo/dt = 
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respectively) do not have any intersections as they do in the static phase where the bump state 
exists. Yet, they are still close enough to create a region with very slow dynamics near the apex 
of the M-nullcline at {u,po) = ^/TjliX - Vf)]. Then, in Fig. [71 we plot the trajectories 

of the dynamics starting from different initial conditions. 

0.06 1 1 
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Figure 7: Trajectories of network dynamics starting from various initial conditions at {k,f3) = 
(0.95, 0.0085) (point P in Fig. [3]). Solid line: u-nullcline. Dashed line: po-nullcline. Symbols 
are data points spaced at time intervals of 2ts. Parameter: t^/ts = 50. 

The most interesting family of trajectories is represented by B and C in Fig. [71 Due to the 
much faster dynamics of u, trajectories starting from a wide range of initial conditions converge 
rapidly, in a time of the order of r^, to a common trajectory in the vicinity of the u-nullcline. 
Along this common trajectory, u is effectively the steady state solution of Eq. (jlOp at the 
instantaneous value of po{t), which evolves with the much longer time scale of r^. This gives rise 
to the plateau region of u which can survive for a duration of the order of r^. The plateau ends 
after the trajectory has passed the slow region near the apex of the n-nullcline. This dynamics 
is in clear contrast with trajectory D, in which the bump height decays to zero in a time of the 
order of r^. 

Trajectory A represents another family of trajectories having rather similar behaviors, al- 
though the lifetimes of their plateaus are not so long. These trajectories start from more depleted 
initial conditions, and hence do not have the chance to get close to the n-nullcline. Nevertheless, 
they converge rapidly, in a time of the order of Tg, to the band u ~ (8/A;)^/^, where the dynamics 
of u is slow. The trajectories then rely mainly on the dynamics of po to carry them out of this 
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slow region, and hence plateaus with lifetimes of the order of are created. 

Following similar arguments, the plateau behavior also exists in the stable region of the 
static states. This happens when the initial condition of the network lies outside the basin of 
attraction of the static states, but still in the vicinity of the basin boundary. 




k 



Figure 8: Contours of plateau lifetimes in the space of k and /3. The lines are the two topmost 
phase boundaries in Fig. [3l In the initial condition, A = 4.82843. 

When one goes deeper into the silent phase, the region of slow dynamics between the u- and 
Po-nullclines broadens. Hence plateau lifetimes are longest near the phase boundary between 
the bump and silent states, and become shorter when one goes deeper into the silent phase. 
This is confirmed by the contours of plateau lifetimes in the phase diagram shown in Fig. [5] 
obtained numerically. The initial condition is uniformly set by introducing an external stimulus 
I^^^{x\zq) = j4exp[— x^/(4a^)] to the right hand side of Eq. ([1]), where A is the stimulus 
strength. After the network has reached a steady state, the stimulus is removed at t = 0, 
leaving the network to relax. It is observed in Fig. [8] that the plateau behavior can be found in 
an extensive region of the parameter space. 

4.2 Second Order: Lifetime Analysis 

As shown in Fig. [6l the first-order perturbation over-estimates the stability of the plateau state, 
yielding lifetimes longer than the simulation results. The main reason is that the width of 
the synaptic depression profile is constrained to be a constant in the first-order perturbation. 
However, the synaptic depression profile is broader than the bump. This can be seen from Eq. 
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([!]), rewritten as 



l-p{x,t) 



Tdf3r{x,t) 
l + Td/3r{x,t) 



(26) 



This shows that the neurotransmitter loss, 1 — p{x,t), relaxes towards an expression consisting 
of the Gaussian r(x,t), normalized by the factor 1 + T(iPr{x,t). This normalization factor is 
smaller where the firing rate is low, so that the profile of 1 — p{x, t) is broader than the firing 
rate profile r{x, t). 

To incorporate the effects of a broadened STD profile, we introduce the second-order pertur- 
bation. Dynamical equations are obtained by truncating the equations beyond the second order. 
As shown in Fig. [U the second-order perturbation yields a much more satisfactory agreement 
with simulation results than do lower order perturbations. 



5 Decoding with Enhanced Accuracy in CANNs with STF 

CANNs have been interpreted as an efficient framework for neural systems implementing popula- 



tion decoding (Deneve, Latham, and Pouget 1999; Wu, Amari, and Nakahara 2002). Consider 



the reading-out of an external feature zq from noisy inputs by CANNs. For example, zq may 
represent the moving direction of an object. In the decoding paradigm, a CANN responds to 
an external input I'^^^{x) with a bump state r{x\z), where the peak position of the bump z is 
interpreted as the decoding result of the network. 

In the presence of STF, neuronal connections are facilitated around the area where neurons 
are most active. With this additional feature, the network decoding will be determined not only 
by the instantaneous input, but also by the recent history of external inputs. Consequently, 
temporal fiuctuations in external inputs are largely averaged out, leading to improved decoding 
accuracies. 

We consider an external input given by 



= Aexp 



{x- zq- r]{t)) 
4a2 



(27) 



where zq represents the true stimulus value, and ri{t) white noise of zero mean and satisfies 
(r?(t)r/(t')) = 2Ta^Ts6{t - t') with T denoting the noise strength. 

In the presence of weak noise, the position of the bump state is found to be centered at 
Zq + s{t), where s{t) is the deviation of the center of mass of the bump from the position of 
stimulus Zq, as derived in Appendix E. Hence, the decoding error of the network is measured by 
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the variance of the bump position over time, namely, (s(t)^). Fig. [9] shows the typical decoding 
performance of the network with and without STF. We see that with STF, the fluctuation of the 
bump position is reduced significantly. Fig. [TO] compares the theoretical and measured decoding 
errors in different noise strengths (see Appendix E). 
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Figure 9: The typical decoding performance of the network with and without STF. Parameters: 
N = 80, a = 0.5, X G [-vr, tt) k = 0.25, A = 1.596 and T = 0.02. 





6 Tracking with Enhanced MobiUty in CANNs with STD 

A key property of CANNs is their capacity to track time-varying stimuli, which lays the foun- 
dation for CANNs to implement spatial navigation, population decoding, and to update head- 
direction memory. To investigate the tracking performance of CANNs, we consider 

(x - Zo {t)f 



4a2 



(28) 



where the stimulus position zq (t) is time dependent. 

We first investigate the impact of STD, and consider a tracking task in which the zq (i) 
abruptly changes from at t = to a new value at t = 0. Fig. [TT] shows the network responses 
during the tracking process. Compared with the case without STD, we find that the bump shifts 
to the new position faster. When f3 is too strong, the bump may overshoot the target before 
eventually approaching it. As remarked previously, this is due to the metastatic behavior of 
the bumps, which enhances their readiness to move from the static state when a small push is 
exerted. 
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Figure 10: The decoding errors of the network vs. different levels of noise. Parameters other 
than T are the same as those in Fig. [9l Symbols: simulations. Dashed line: predictions for 
a = 0. Solid line: predictions for a = 0.1. 

We also study the tracking of an external stimulus moving with a constant velocity v, that is, 
zq (t) changes from to ft at t = 0. As shown in Fig. HSJa), when STD is weak, the initial speed 
of the bump is almost zero. Then, when the stimulus is moving away, the bump accelerates in 
an attempt to catch up with the stimulus. After some time, the separation between the bump 
and the stimulus converges to a constant. This tracking behavior is similar to the case without 
STD. The tracking behavior in the case of strong STD is more interesting. As shown in Fig. 
[T2r b). the bump position eventually overtakes the stimulus, displaying an anticipative behavior. 
This can be attributed to the metastatic property of STD. 

We further explore how STF affects the tracking performance of CANNs. In general, there is 
a trade-off between the stability of network states and the capacity of the network to track time- 
varying stimuli. Since STD and STF have opposite effects on the mobility of the network states, 
we expect that they will also have opposite impacts on the tracking performance of CANNs. 
Indeed, STF degrades the tracking performance of CANNs (see Fig. [T3l) . The larger the STF 
strength, the slower the tracking speed of the network. 

7 Conclusions and Discussions 

In the present study, we have investigated the impact of STD and STF on the dynamics of 
CANNs and their potential roles in neural information processing. We have analyzed the dy- 
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Figure 11: The response of CANNs with STD to a stimulus that changed abruptly from 
ZQ/a = ^ to Z{)/a = 3.0 at t = 0. Symbols: numerical solutions. Lines: Gaussian approximation 
using 11*^ order perturbation of the STD coefficient. Parameters: Td/rg = 50, A = 4.82843, 
AT = 80, a = 0.5 and x E [-vr, vr). 

namics using successive orders of perturbation. The perturbation analysis works well when STD 
is not too strong, although it over-estimates the stability of the bumps when STD is strong. 
The zeroth order analysis accounts for the Gaussian shape of the bump, and hence can predict 
the boundary of the static phase satisfactory. The first-order analysis includes the displacement 
mode and asymmetry with respect to the bump peak, and hence can describe the onset of the 
moving phase. Furthermore, it provides insights on the metastatic nature of the bumps and its 
relation with the enhanced tracking performance. The second-order analysis further includes 
the width distortions, and hence improves the prediction of the boundary of the moving phase, 
as well as the lifetimes of the plateau states. Higher-order perturbations are required to yield 
more accurate descriptions of results such as the overshooting in the tracking process. We an- 
ticipate that the perturbation analysis will also be useful in many other population decoding 
problems, such as in quantifying the deformation of tuning curves due to neural adaptation 
( Cortes, Marinazzo, Series, Oram, Sejnowski, and van Rossum 2011 ). 

More importantly, our work reveals a number of interesting behaviors which may have far- 
reaching implications in neural computation. 

First, STD endows CANNs with slow-decaying behaviors. When a network is initially stim- 
ulated to an active state by an external input, it will decay to the silent state very slowly after 
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Figure 12: Tracking of a neuronal bump with a continuously moving stimulus. Symbols: the 
peak of the bump from simulation. Dashed line: 11*^ order perturbation prediction. Solid line: 
a continuously moving stimulus with speed Tgv/a = 0.06. Parameters: A = 1.5958, k = 0.5. (a) 
j3 = 0.01. (b) /? = 0.05. Other parameters are the same as those in Fig. [TTJ 



the input is removed. The duration of the plateau is of the time scale of STD rather than of 
neural signaling. This provides a way for the network to hold the stimulus information for up 
to hundreds of milliseconds, if the network operates in the parameter regime where the bumps 
are marginally unstable. This property is, on the other hand, extremely difficult to implement 
in attractor networks without STD. In a CANN without STD, an active state of the network 
will either decay to the silent state exponentially fast or be retained forever, depending on the 
initial activity level of the network. Indeed, how to shut off the activity of a CANN gracefully 
has been a challenging issue that has received wide attention in theoretical neuroscience, with 
researchers suggesting that a strong external input either in the form of inhibition or excitation 
must be applied (see, e.g., Gutkin, Laing, Colby, Chow, and Ermentrout (2001)). Here, we have 
shown that in certain circumstances, STD can provide a mechanism for closing down network 
activities naturally and after a desirable duration. Taking into account the time scale of STD 
(in the order of 100 ms) and the passive nature of its dynamics, the STD-based memory is most 
likely associated with the sensory memory of the brain, e.g., the iconic and the echoic memories 



dBaddeley 1999D . 

Second, with STD, CANNs can support both static and moving bumps. Static bumps exist 
only when the synaptic depression is sufficiently weak. A consequence of synaptic depression is 
that static bumps are placed in the metastatic state, so that its response to changing stimuli 
is speeded up, enhancing its tracking performance. The states of moving bumps may be as- 
sociated with the travelling wave behaviors widely observed in the neurocortex. We have also 
observed that for strong STD, the network state can even overtake the moving stimulus, rem- 



iniscent of the anticipative responses of head-direction and place cells (Blair and Sharp 1995 
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lO'Keefe and Recce 1993"]) . It is interesting to note that this occurs in the parameter range 
where the network holds spontaneous moving bump solutions, suggesting that travelling wave 
phenomena may be closely related to the predicting capacity of neural systems. 

Third, STF improves the decoding accuracy of CANNs. When an external stimulus is 
presented, STF strengthens the interactions among neurons which are tuned to the stimulus. 
This stimulus-specific facilitation provides a mechanism for the network to hold a memory trace 
of external inputs up to the time scale of STF, and this information can be used by the neural 
system for executing various memory-based operations, such as operating the working memory. 
We have tested this idea in a population decoding task, and found that the error is indeed 
decreased. This is due to the determination of the network response by both the instantaneous 
value and the history of external inputs, which effectively averages out temporal fluctuations. 

These computational advantages of dynamical synapses lead to the following implications 
for the modeling of neural systems. First, it sheds some light on the long-standing debate in 
the field about the instability of CANNs in the presence of noise. Two aspects of instabil- 
ity have been identified (IWu and Amari 2005t Seung, Lee, Reis, and Tank 2000 ). One is the 
structural instability, which refers to the argument that network components in reality, such 
as the neuronal synapses, are unlikely to be as perfect as mathematically required in CANNs. 
A small amount of discrepancy in the network structure can destroy the state space consider- 
ably, destabilizing the bump state after the stimulus is removed. The other instability refers to 
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the computational sensitivity of the network to input noises. Because of neutral stability, the 
bump position is very susceptible to fluctuations in external inputs, rendering the network de- 
coding unreliable. We have shown that STF can largely improve the computational robustness 
of CANNs by averaging out the temporal fluctuations in inputs. Similarly, STF can overcome 
the structural susceptibility of CANNs. With STF, the neuronal connections around the bump 
area are strengthened temporally, which effectively stabilizes the bump on the time scale of 



STF (Tsodyks and Hansel 2011). Another mechanism in a similar spirit is the reduction of the 



inhibition strength around the bump area (Carter and Wang 2007). 

Second, STD and STF should be dominant in different areas of the brain. We have investi- 
gated the impact of STD and STF on the tracking performance of CANNs. There is, in general, 
a trade-off between the stability of bump states and the tracking performance of the network. 
STD increases the mobility of bump states and hence the tracking speed of the network, whereas 
STF has the opposite effect. These differences predict that in cortical areas where time-varying 
stimuli, such as the head direction and the moving direction of objects, are encoded, STD should 
have a stronger effect than STF. On the other hand, in cortical areas where the robustness of 
bump states, i.e., the decoding accuracy of stimuli, is preferred, STF should have a stronger 
effect. 

Third, STD and STF consume different levels of energy and operate on different time scales. 
We have shown that both STD and STF can generate temporal memories, but their ways of 
achieving it are quite different. In STD, the memory is held in the prolonged neural activities, 
whereas in STF, it is in the facilitated neuronal connections. Mongillo, Barak, and Tsodyks 
(2008) proposed that with STF, neurons may not even have to be active after the stimulus is 
removed. The facilitated neuronal connections, mediated by the elevated calcium residue, is 
sufficient to carry out the memory retrieval. In our model, this is equivalent to setting the 
network in the parameter regime without static bump solutions, or in the regime with static 
bump solutions but with the external stimulus presented for such a short time that neuronal 
interactions cannot be fully facilitated. Thus, taking into account the energy consumption asso- 
ciated with neural firing, the STF-based mechanism for short-term memory has the advantage 
of being economically efficient. However, the STD-based one also has the desirable property of 
enabling the stimulus information to be propagated to other cortical areas, since neural firing is 
necessary for signal transmission, and this is critical in the early information pathways. Further- 
more, the time durations required for eliciting STF- and STD-based memory are significantly 



24 



different. The former needs a stimulus to be presented for an amount of time up to r/ to facili- 
tate neuronal interactions sufficiently; whereas the latter simply requires a transient appearance 
of a stimulus. This difference implies that the two memory mechanisms may have potentially 
different applications in neural systems. 

In summary, we have revealed that STP can play very valuable roles in neural informa- 
tion processing, including achieving temporal memory, improving decoding accuracy, enhancing 
tracking performance and stabilizing CANNs. We have also shown that STD and STF tend 
to have different impacts on the network dynamics. These results, together with the fact that 
STP displays large diversity in the neural cortex, suggest that the brain may employ a strategy 
of weighting STD and STF differentially depending on the computational task. In the present 
study, for the simplicity of analysis, we have only explored the effects of STD and STP sepa- 
rately. In practice, a proper combination of STD and STP can make the network exhibit new 
and interesting behaviors and implement new and computationally desirable properties. For 
instance, a CANN with both STD and STF, and with the time scale of the former shorter than 
that of the latter, can hold bump states for a period of time before shifting the memory to 
facilitated neural connections. This enables the network to achieve both goals of conveying the 
stimulus information to other cortical areas and holding the memory cheaply. Alternatively, 
the network may have the time scale of STD longer than that of STF, so that the network 
can produce improved encoding results for external stimuli and also close down bump activities 
easily. We will explore these interesting issues in the future. 
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Appendix A: Consistency with the Model of Tsodyks et al 

Our modeling of the dynamics of STP is consistent with the phenomenological model proposed 
by Tsodyks, Pawelzik, and Markram (1998). They modeled STD by considering p as the frac- 
tion of utilizable neurotransmitters, and STF by introducing Uq as the release probability of 
the neurotransmitters. The release probability Uq relaxes to a nonzero constant, Urest) but is 
enhanced at the arrival of a spike by an amount equal to uq[1 — Uq). Hence the dynamics of p 
and Uq are given by 

du f 
Ts— = Icxt - u + p dx'J{x - x')p{x')Ui{x')r{x'), (A-1) 

dp 

Td-Q^ = l-p-UiTdpr, (A-2) 
OU 

'^f-gf = ""rest -Uo+ Uq{1 - UQ)Tfr, (A-3) 

where Ui = uq + Uq{1 — uq) is the release probability of the neurotransmitters after the arrival 
of a spike. The x and t dependence of u, p, r and Uq are omitted in the above equations for 
convenience. Eliminating Uq, we obtain from Eq. ()A-3P 

dUi Uq + {1 - Uq) Urest - Ui , jj. 

— - = huo(l-f7i)r. (A-4) 

dt Tf 

Substituting a, /? and / via a = uq, f3 = uq + {1 - UQ)urest, Ui = [uq + (1 - uo)iircst](l + /), 
/max = — P)//3, we obtain Eqs. (HD and Rescaling 13 J to J, we obtain Eq. ([T]). a and f3 
are the STF and STD parameters respectively, subject to (3 > a. 

Appendix B: The Perturbation Approach for Solving the Dynam- 
ics of CANNs with STD 

We use the perturbation approach to solve the network dynamics. Substituting Eqs. (|lUp and 
([13]) into Eq. ([TJ, the right-hand side of Eq. ([T|) becomes 



P_ 
B 



^a„am / dx'j{x,x')vn{x',t)vm{x',t) 

nm 

-'^ttnambl j dx'j{x,x')Vn{x',t)Vmix',t)wi{x',t) 
nml 

-Y,an{t)Vn{x,t) + P^\x,t). (B-1) 
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This expression can be resolved into a linear combination of the distortion modes Vk{x,t). The 
coefficients of these modes are obtained by multiplying the expression with Vk{x,t) and inte- 
grating X. Using the orthonormal property of the distortion modes, we have 

pC-r-Jo 



B 



Xnm nml / 



where 



j J dx'j{x",x')vk{x",t)Vn{x',t)Vm{x',t), 
'^nml = / I dx'j{x",x')vk{x",t)Vn{x',t)Vm{x',t)wi{x',t), 



(B-2) 

(B-3) 
(B-4) 



and Ik{t) is the k^^ component of P^^{x,t). 

Similarly, the right-hand side of Eq. @ becomes 



'^Wk{x,t) 



dx'wkix', t)Vn{x', t)Vmix', t), 



(B-5) 

fe L V nm nml / . 

where 

(B-6) 
(B-7) 

We choose Vn{x,t) = Vnix — z{t)) and Wn{x^t) = Wn{x — z{t)). Using the following relationship 
of Hermite polynomials 



^nml = / dx'wk{x',t)Vn{x',t)Vm{x',t)wi{x',t). 



Hn+i{x) = 2xHnix) - 2nHn-iix), 
H'^{x) = 2ni7„„i(x), 



(B-8) 
(B-9) 



we have Vn = {z / {2a)){^/nvn-i - ^/n + Ivn+i) and Wn = {z / {\/2a)){'s/nWn-i - ^/n + \wn+i)- 
Making use of the orthonormality of w„'s and Wn's, we have 



Ofc - ^ (^^/k + lofc+i - Vkak-i 



-ak + 



pCrJi. 







B 



(B-10) 



nml 



Td 



bk 



V2a 



Vk + Ibk+i - Vkhk-i 



B 



nm^n^m 



■SnmlO'nO'mbl j • 



nml 



(B-11) 



The values of D^^;, and 

'^nmi '^^^ obtained from recurrence relations derived 
using integration by parts and the relationships ()B-8P and (|B-9p . which are given by 



= — I /— C'^i -I- /_(J*'^i 
*-nm 2 V V k y k ' ' 



(B-12) 
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C =--4/— 

4 V "2"" 4 V n "i""! 2 V n "1"*' 

4 V m 4 V m "I"'! 2 V m^"*i ' 

where ni = n — 1 and m2 = m — 2 etc. in the indices. Similarly, 

-'^ ( / 't — i., / /'(- — t., /:r ; (- _ i., \ 1 



nml 



rr\k 



'^nml 



nm 
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B-13) 
B-14) 

B-15) 

B-16) 

B-17) 
B-18) 
B-19) 
B-20) 
B-21) 
B-22) 

B-23) 

B-24) 

B-25) 



Since Cgo, 2)ooo> "^oo and g'ooo can be calculated explicitly, all other C^^, 2)^^^, and 5^^, 
can be deduced. 

Below, we analyze the dynamics of the bump in successive orders of perturbation, where the 
perturbation order is defined by the highest integer value of index k involved in the approxima- 
tion. We start with the zeroth order perturbation to describe the behavior of the static bumps, 
since their profile is effectively Gaussian. We then move on to the first-order perturbation, which 
includes asymmetric distortions. Since spontaneous movements of the bumps are induced by 
asymmetric profiles of the synaptic depression, we demonstrate that the first-order perturbation 
is able to provide the solution of the moving bump. Proceeding to the second-order perturba- 
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tion, we allow for the flexibility of varying the width of the bump and demonstrate that this is 
important in explaining the lifetime of the plateau state. Tracking behaviors are predicted by 
the 11*^ order perturbation. 



Appendix C: Static Bump: Lowest-Order Perturbation 

Without loss of generality, we let z = 0. Substituting Eqs. (fTT]) and into Eq. ([T]), we get 



dt \/2 (l + V^Trakpul) 



e ia^ -P0\l-e 3a^ 



uoe + Ae 3^. (C-1) 



Using the projection onto vq, we find that exp (— x^/Sa^) ~ y^6/7 exp (— x^/4a^). This reduces 
the equation to 

dUQ ^ pCrJoUQ 



-uo + 



dt " ' y/2{l + V2^akpul) 

Introducing the rescaled variables u and k, we get Eq. (fl9l) . 
Similarly, substituting Eqs. (fT7|) and ([T8|) into Eq. gives 
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dt ' 1 + y/2TTakpul 

Making use of the projection exp [—x^ /a^) ^ a/273 exp 

/2a^), the equation simplifies to 



dpo , TdPu^ 

dt 1 + yj2'nakpu'^ 



1 
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Rescaling the variables u, k, (5 and A, we get Eq. (j20|) . 

The steady state solution is obtained by setting the time derivatives in Eqs. (jl9p and (j20 
to zero, yielding 
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where B = 1 + ku^ /8 is the divisive inhibition. 

Dividing Eq. (1C-5P by (IC-6p . we eliminate B and get 
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We can eliminate u from Eq. ()C-6P . This gives rise to an equation for po 
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2/3 I 



2 k\ 

3 + ^P° 



Rearranging the terms, we have 



A; = — 1 
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Therefore, for each fixed pq, we can pfot a parabohc curve in the space of j3 versus k. The dashed 
hues in Fig. [14] are parabolas for different values of pq. The family of all parabolas map out the 
region of existence of static bumps. 



Stability of the Static Bump 

To analyze the stability of the static bump, we consider the time evolution of e = u{t) — u* and 
^ = Po (t) ~Po' where {u*,Pq) is the fixed point solution of Eqs. (IC-SP and (lC-6h . Then, we have 




(C-10) 



Ase Ass 

where A,, = -l/Ts+V2u{l-y^po)/TsB^ A,s = -u^y^/TsB, As, = 2Bu{l- ^/ij^po) / t^B^ , 
Ass = —^/Td — P^'^ \/ / 3 / T^B . The stability condition is determined by the eigenvalues of the 




stability matrix, [T ± ^T^ - 4Dj /2, where D and T are the determinant and the trace of the 
matrix respectively. Using Eqs. (]C-5p and ()C-6p . the determinant and the trace can be simplified 
to 

^^^(jvmn 
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Td{l- V2/3po^ 

The static bump is stable if the real parts of the eigenvalues are negative. The eigenvalues 
are real when > AD. This corresponds to non-oscillating solutions. After some algebra, we 
obtain the boundary = AD given by 
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Figure 14: The region of existence of static bump solutions. Solid line: the boundary of existence 
of static bump solutions. Dashed lines: the parabolic curves for different constant values of po- 



This boundary is shown in Fig. [T5j Below this boundary, the stability condition can be obtained 
as 



Poll- V^Pi 



(C-14) 



^/2/^po) [I - 274/7^0 + V8/21pl^ 
This upper bound is identical to the existence condition of (jC-Qh . which is above the boundary 
of non-oscillating solutions. This implies that all non-oscillating solutions are stable. 

Above the boundary (|C-13p . the convergence to the steady state becomes oscillating, and 
the stability condition reduces to T < 0, yielding Eq. (j21|) . This condition narrows the region 
of static bump considerably, as shown in Fig. [T5l 



Appendix D: Moving Bump: Lowest-Order Perturbation 

We substitute Eqs. ([22]) and (p3]) into Eqs. ^ and Eq. (1) becomes an equation contain- 



mg exp 



and exp 



(x — vt)'^ /4a^ (x — vt) /a, after making use of the pro- 



jections exp[— (x — ft)^/3a^] y^6/7 exp[— (x — ft)^/4a^], and exp[— (x — ft)^/3a^](x — vt)/a 



(v/677)^exp[-(x - vtf/Aa^jix - vt)/a. 

Equating the coefficients of exp — {x — vtf' / Aa? 
rescaling the variables, we arrive at 
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Figure 15: The region of stable solutions of the static bump for t^It^ = 50. Solid line: the 
boundary of stable static bumps. Dashed line: the boundary separating the oscillating and 
non-oscillating convergence. Dotted lines: the curves for different constant values of pQ. 



Similarly, making use of the projections exp[—(3;— ft) /a ] « Y^2/3exp[— (x— wt) /2a ], exp[— (x- 
vtf /a?]{x - vt)/a w (y^2/3)^exp[-(a; - vt)^/2a^](x - vt)/a, we find that Eq. (jl]) gives rise to 
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The solution can be parametrized by ^ = f3u'^ /B 
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Po 



G(0 



^= 2^F{0, PI 
a V Td 



(D-4) 



where F(0 = (4/7)3/2^ - (r,/r,)[l + {2/3f/^C][l - (v^ - and G(0 = (4/7)3/2 + 

(4:/7y/'^{Ts/Td)[l + (2/3)3/2^]_ A real solution exists only if Eq. ^ is satisfied. The solution 
enables us to plot the contours of constant ^ in the space of k and /?. Using the definition of ^, 
we can write 

I = f/J-^(|)'/^^ 

where the quadratic coefficient can be obtained from Eq. ()D-4p . Fig. [16] shows the family of 
curves with constant ^, each with a constant bump velocity. The lowest curve saturates the 
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inequality in Eq. (13), and yields the boundary between the static and metastatic or moving 
regions in Fig. [3l Considering the stability condition in the next subsection, only the stable 
branches of the parabolas are shown. 



Stability of the Moving Bump 



To study the stability of the moving bump, we consider fluctuations around the moving bump 
solution. Suppose 
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(D-6) 
(D-7) 



These expressions are substituted into the dynamical equations. The result is 
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We first revisit the stability of the static bump. By setting v and pi to 0, considering the 
asymmetric fluctuations s and ei in Eqs. (ID-9h and (jD-llh . and eliminating ds/dt, we have 



2u /2\ 2 



ei. 



(D-12) 



- — 

dt [B \7j Td 
Hence the static bump remains stable when the coefficient of ei on the right-hand side is non- 
positive. Using Eq. (jD-4p to eliminate po and u/B, we recover the condition in Eq. (fM|l . This 
shows that the bump becomes a moving one as soon as it becomes unstable against asymmetric 
fluctuations, as described in the main text. 

Now we consider the stability of the moving bump. Eliminating ds/dt and summarizing the 
equations into matrix form, 

3^/7 ^ 
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Figure 16: The stable branches of the family of the curves with constant values of at t^/ts = 50. 
The dashed line is the phase boundary of the static bump. 

where Pqu = 2poTs/ BuTd+vTsPi/2au, Pqq = VTsPi/2apQ- pu^Ts/ Bp^T^, and Piu = 2piTs/BuTd- 
VTsPo/au. For the moving bump to be stable, the real parts of the eigenvalues of the stability 
matrix should be non-positive. The stable branches of the family of curves are shown in Fig. [TBI 
The results show that the boundary of stability of the moving bumps is almost indistinguishable 
from the envelope of the family of curves. Higher-order perturbations produce phase boundaries 
that agree more with simulation results, as shown in Fig. [3l 

We compare the dynamical stability of the ansatz in Eqs. (j22p and (j23p with simulation 
results. As shown in Fig. [T71 the region of stability is over-estimated by the ansatz. The major 
cause of this is that the width of the synaptic depression profile is restricted to be a. While 
this provides a self-consistent solution when STD is weak, this is no longer valid when STD 
is strong. Due to the slow recovery of synaptic depression, its profile leaves a long, partially 
recovered tail behind the moving bump, thus reducing the stability of the bump. This requires 
us to consider the second-order perturbation, which takes into account variation of the width of 
the STD profile. As shown in Fig. [T71 the second-order perturbation yields a phase boundary 
much closer to the simulation results when STD is weak. However, as shown in the inset of Fig. 
[T71 the discrepancy increases when STD is stronger, and higher-order corrections are required. 
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IcQ- 




Figure 17: The boundary of the moving phase. Symbols: simulation results. Dashed line: l'^* 
order perturbation. Solid line: 2"^^^ order perturbation. Inset: The boundary of the moving 
phase in a broader range of (3. Parameters: N/L = 80/(27r), a/L = 0.5/(27r). 

Appendix E: Decoding in CANNs with STF 



We start by considering bumps and STF profiles of the form 

(x - zo - s{t)f 



u{x, t) = uo exp 
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Note that since the noise occurs in the position of the bump, we can neglect changes in the 
height. Substituting Eqs. (lE-ip and (lE-ip into Eq. ([T]), and removing terms orthogonal to the 
position distortion mode, we have 
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This differential equation can be solved by first diagonalizing M. Let —\± be the eigenvalues 
of M, and {Us± Uf±)^ be the corresponding eigenvectors. Then the solution becomes 
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where E± = exp[— A±(t — ti)]. Squaring the expression of s/a, averaging over noise, and inte- 
grating, we get 



a,b=± 
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Us, (u,-'-U,-/)].{E-Q) 
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